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Abstract 

Taking a college entrance exam such as the 
ACT® or SAT® can be an important step in 
the college-going process. Although overall, 

1 out of 1 0 students who register for the ACT 
on a national test date are absent on their 
registered test day, the absentee rate (at 
21 %) for students who registered with a fee 
waiver is substantially higher than the 
absentee rate (at 5%) for students who paid 
the registration fee. The current study 
examined whether a brief telephone 
reminder of the test date and of the required 
test materials a few days prior to the test 
date would result in fee waiver students 
having higher test day attendance rates. 
Using a randomized controlled trial with 
2,500 students, the findings of this study do 
not provide evidence that the treatment as 
delivered had an impact on the attendance 
rates of fee waiver students. There are 
several possible rationales for why the 
treatment was not effective, and these 


rationales point to areas of future direction 
for this research initiative. 

Introduction 

Taking a college entrance exam such as the 
ACT or SAT can be an important step in the 
college-going process. According to the most 
recent data from the National Center for 
Education Statistics (2015), 1,644 
postsecondary educational institutions 
across the US serving roughly eight million 
degree-seeking undergraduate students 
either require or recommend test scores for 
admissions decisions. Sending an official 
ACT or SAT score report to a postsecondary 
institution as part of the application process 
satisfies this criterion. ACT sent over six 
million official score reports to postsecondary 
institutions on behalf of the ACT-tested high 
school graduating class of 2014 
(N=1 ,845,787) — an average of 3 score 
reports per student (ACT, 2015). 
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Taking the ACT or SAT can also expand the number and type 
of postsecondary institutions that students consider during the 
search stage of the college choice process by providing an 
opportunity for students to be identified and recruited by 
institutions that are not currently under consideration. At the 
time that students register for the ACT or SAT, they may opt 
into a service such as ACT’s Educational Opportunity Service 
(EOS) or The College Board’s Student Search Service. These 
services allow accredited postsecondary institutions to identify 
and select the names and contact information (i.e., email 
address or street address) of prospective students whose 
backgrounds and academic achievement levels align with the 
mix of desired characteristics of the students in their applicant 
pools. For instance, about 86% of the ACT-tested graduating 
class of 2014 opted into EOS, with over 1,000 postsecondary 
institutions using EOS to identify and select these students. 
Overall, these students had their contact information obtained 
by an average of 17 postsecondary institutions that they were 
not previously considering. About 16% of these students 
subsequently enrolled at an institution that selected them 
through EOS (ACT, 2015). 

Although both ACT and The College Board have contracts with 
a limited number of states to offer the ACT or SAT statewide, 
most college-bound students continue to take either of these 
exams on one of six to seven national test dates that are 
offered on Saturdays during the school year. The current fee to 
sit for the ACT or SAT is $39.50 ($56.50 with Writing) and $43 
($54.50 with Essay), respectively. These fees may create a 
financial burden for lower-income students and their families, 
thus potentially limiting their access to many postsecondary 


institutions across the US. To offset this financial burden and to 
increase test-taking and subsequent college attendance 
among lower-income students, both ACT and The College 
Board have created programs that provide income-eligible 
students with fee waivers in order to register to take their 
respective test on a national test date. These fee waiver 
programs are large in scope and well-utilized, accounting for a 
sizable share of the total test registrations per school year. For 
example, 29% of all ACT test registrations for the 2014-15 
school year were completed using a fee waiver. This amounts 
to $29.2 million offered in fee waivers, covering over 600,000 
ACT registrations. 

Within both ACT and The College Board’s fee waiver 
programs, eleventh or twelfth grade students who are either 
enrolled in or eligible to participate in the National School 
Lunch Program may obtain up to two fee waivers to use for 
national test day registration. 1 The students’ eligibility for a fee 
waiver is verified by a high school counselor or other 
designated school official and the fee waiver is distributed 
locally. Each fee waiver then applies to the registration for an 
individual test date. If the student does not attend on their 
registered test date, the fee waiver is forfeited and cannot be 
applied to a future test date. 

Despite the importance of taking a college entrance exam as a 
step toward increasing postsecondary educational opportunity, 
a large and disproportionate share of students who register 
with a fee waiver do not subsequently attend on their 
registered test date. For example, although overall, 1 out of 10 
students who registered for the ACT on a national test date 
during the 2013-14 school year were absent on their registered 
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test date, the absentee rate for students who registered with a 
fee waiver (at 21%) is substantially higher than the absentee 
rate for students who paid the registration fee (at 5%). 

To address this discrepancy in test day attendance rates, ACT 
announced a new research initiative, known as “Dialing for 
Scholars,” at the White House Summit on College Opportunity 
in December, 2014 (ACT, 2014). This initiative was designed 
with the goal of increasing the test-taking rate for students who 
registered with fee waivers by reaching out to these students 
via phone or email in the weeks before the scheduled national 
ACT test dates to proactively share information that would 
better prepare them for the testing experience and to 
encourage them to attend. 2 

Numerous studies from the health sciences that employ 
randomized controlled trials have found that reminders have a 
positive impact on engaging in various health-related behaviors 
(e.g., Armstrong, et al., 2009; Fry & Neff, 2009; Hurling, et al., 
2007; Lantz, et al., 1995; Liang, et al., 2011; Patrick, et al., 
2009; Pop-Eleches, et al., 2011; Stockwell, et al., 2012), 
including attendance at healthcare appointments (Gurol- 
Urganci, de Jongh, Vodopivec-Jamsek, Atun & Car, 2013). 
Although utilized less frequently in the area of education, a few 
recent randomized controlled trials (e.g., Castelman & Page, 
2015, 2016) have found that reminders are an effective means 
of getting students to follow through on the pre-matriculation 
(e.g., registering for orientation and completing housing forms) 
and post-matriculation (e.g., re-filing the Free Application for 
Federal Student Aid) tasks that in turn lead to higher college 
enrollment and retention rates, respectively. 


The current study was developed as a small pilot experiment to 
establish a baseline for future efforts under the “Dialing for 
Scholars” initiative. The question of interest addressed in this 
pilot experiment is: Does a brief reminder of the test date and 
of the required test materials a few days prior to the test date 
result in fee waiver students having higher test day attendance 
rates? The treatment for this study entailed sending a pre- 
recorded informational telephone message to a random 
sample of students a few days prior to the national test date in 
which they had registered with a fee waiver. This pilot design 
serves as a baseline, as it requires the least amount of 
deviation from current ACT business practices regarding the 
content, mode, and timing of contacts with students for other 
test registration and reporting processes. 

Methods 

Design 

The design for this study was a randomized controlled trial 
(RCT) with an intention-to-treat (ITT) analysis. By randomly 
assigning students to the intervention, an RCT makes sure 
that, on average, there is balance (i.e., no underlying 
differences) between the treatment and control groups, and 
this in turn increases confidence that any between-group 
difference in the outcome is attributable to the intervention. ITT 
analyses include every student who was selected to receive 
the treatment regardless of whether or not that student 
complied with the treatment, received a deviation in the 
treatment, or withdrew from the treatment. With an ITT 
approach to analysis, the research question of interest is the 
effectiveness of prescribing the treatment — its “use- 
effectiveness” — as opposed to receiving the treatment — its 
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“method-effectiveness” (Sheiner& Rubin, 1995). In other 
words, the results of this research will provide some evidence 
of the impact of a pre-recorded telephone reminder on the test 
day attendance of fee waiver students knowing that not all fee 
waiver students would necessarily receive and subsequently 
act on that reminder. 

Treatment 

Content . The treatment for this study comprised a pre-recorded 
telephone message. The content of the message — which 
provided the recipient with a reminder of the test date and of 
the materials that the student is required to bring on test day — 
was written to be informational in nature and free of any 
language about the benefits of attending (e.g., increasing 
college opportunity) or the costs of not attending (e.g., 
forfeiting one of only two fee waivers). Specifically, the 
message stated: “This is a call from [insert state-specific ACT 
legal name] calling to remind you that you have registered to 
take the ACT on Saturday, April 18. Please report to your 
assigned test center by 8:00am, and be sure to bring a copy of 
your ticket and an acceptable photo identification with you to 
your test center. If you have any questions about your test day, 
you can contact us at 319-337-1270 between 8:00am-8:00pm 
CT Monday-Friday or email us at ACT-Reg@act.org. Please 
press 1 to hear this message again.” 


submit their photo identification as part of the registration 
process are submitted by a pre-recorded message a few 
weeks before each national test date. 

Timing . The Thursday just prior to the national test date was 
selected for administering the treatment. This particular date 
was chosen, as the deadline for submitting all necessary test 
registration materials to ACT was set at one week prior to the 
test date, and meeting this deadline was a criterion for entry 
into the target population for the study. A standardized protocol 
for contacting the students was followed that allowed for up to 
two attempts to reach the students. 


Variables and Analysis 

The dependent variable in this study is student attendance at 
their designated test center during the April 18, 2015 national 
test date. Operationally, the dependent variable is 
dichotomous, where: 


yi = 


1 


1 attended on national test date 
0 did not attend on national test date 


The independent variable in this study is the students’ 
membership in the treatment group or control group, 
regardless of whether students in the treatment group received 
the pre-recorded telephone message. Operationally, the 
independent variable is also dichotomous, where: 


Mode . The treatment was delivered the Thursday (i.e., two 
days) prior to the test date via computerized autodialer to the 
telephone number that the student provided on the ACT 
registration form. This mode of delivery was chosen as it is a 
common mode in which ACT communicates to students prior 
to the test date. For example, reminders that students must 



1 randomly assigned to treatment group 
0 randomly assigned to control group 


Given the categorical nature of the dependent and 
independent variables, the test day attendance frequencies for 
each group were arranged in a 2X2 contingency table and a 
Pearson Chi-square test for two proportions was calculated 
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using the FREQ procedure in SAS. The null hypothesis for this 
study is that the treatment and control groups have equivalent 
test day attendance rates: 

Ho- Pt — Pc = 0- 

The alternative hypothesis is that the test day attendance rate 
for the treatment group is larger than the rate for the control 
group: 

Hi- Pt — Pc > 0- 

Sample 

The population for this study comprised students who had 
registered for ACT’s Saturday, April 18, 2015 national test date 
with a fee waiver. For the prior three years (i.e., 2012, 2013, 
and 2014) the April test day attendance rates among fee 
waiver students were consistently around 78%. For this study, I 
wanted to be able to detect a five percentage-point increase in 
the attendance rate (i.e., 78% to 83%) due to receiving the 
treatment with a Type 1 error rate (a) of 0.05 and with a power 
(1 - P) exceeding 0.80. Given that this was intended to be a 
small pilot and I wanted to minimize the possibility of 
contamination (e.g., students in the treatment group 
communicating with students in the control group about the 
phone reminder), I opted for an unbalanced sampling ratio of 
1:4 between the treatment and control groups. Given these 
criteria, 500 fee waiver students from the population were 
randomly selected and assigned to receive the treatment and 
2,000 were randomly selected and assigned to the control 
group. 


Descriptive statistics for the self-reported background 
characteristics of the treatment and control groups are 
provided in Table 1 . As seen in the table, random assignment 
did a fairly robust job of balancing the observable 
characteristics of the students in these two groups. However, 
compared to students in the control group, those in the 
treatment group were slightly less likely to aspire to earn a 
graduate-level degree. Students in the treatment group were 
also slightly less likely to be white and slightly more likely to 
have parents with no college education. 

Results 

An initial telephone contact was attempted for 497 out of the 
500 students in the treatment group. Among the three students 
not contacted, one student had provided an invalid telephone 
number and two students had telephone numbers with area 
codes that were outside of the 50 states or the District of 
Columbia. Among the 497 students initially contacted, 441 
were successfully contacted on the first attempt. A successful 
contact was defined as either a live answer or a redirect to 
voice mail. A second attempt was made an hour later for 
students where the initial attempt at contact was unsuccessful. 
The second attempt yielded eight more successful contacts. 

In all, 449 of the 500 students who were randomly assigned to 
the treatment group received the pre-recorded message, 173 
of the calls were designated as live answer, and the other 276 
calls were designated as having been redirected to voice mail. 
The median duration for both live answer and answering 
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machine redirects was 52 seconds, 
suggesting that many individuals who 
participated in the live answer listened 
to the message in its entirety before 
terminating the call. 

Among the 2,500 students sampled for 
the experiment, 2,013 students 
(80.5%) attended on the national test 
date, whereas 487 students (19.5%) 
were absent. Table 2 provides the 
attendance rates for the treatment and 
control groups. As seen in Table 2, 
only 78.6% of students from the 
treatment group attended on the test 
date, compared with 81.0% of students 
from the control group. This difference, 
which is not in the hypothesized 
direction, is not statistically significant 
from zero. These results support the 
null hypothesis of no difference in test 
day attendance rates between the 
students in the two groups. 

Discussion and Future 
Directions 


Table 1. Background Characteristics of Treatment and Control Groups 


Outcome 

Treatment Group 

Control Group 

High School GPA 

3.16 (0.59) 

3.15 (0.59) 

Degree Expectations 

Less than bachelor's degree 

7 

7 

Bachelor's degree 

45 

46 

Graduate degree 

30 

32 

Missing 

17 

15 

Gender 

Female 

42 

43 

Male 

58 

57 

Race/Ethnicity 

African American 

29 

28 

American Indian 

3 

2 

Asian 

4 

4 

Hispanic 

31 

30 

White 

24 

26 

Other race 

9 

9 

Parents' Education 

No college 

36 

34 

Some college 

27 

27 

Bachelor's degree 

15 

14 

Graduate degree 

5 

6 

Missing 

17 

19 

N 

500 

2000 


Table 2. Test Day Attendance by Study Group 

Outcome 

Treatment Group 

Control Group 

Present 

393 (78.6%) 

1620 (81.0%) 

Absent 

107 (21.4%) 

380 (19.0%) 


Notes: 1) Pearson Chi-square = 1.4689 (1 df). 2) N = 2500 


The purpose of this study was to 
examine whether a brief reminder of 
the test date and of the required test 
materials increases the test day 
attendance rate of fee waiver students. 


The findings of this pilot experiment do 
not provide evidence that the 
treatment as delivered had an impact 
on the attendance rates of fee waiver 
students. There are several possible 


rationales for why the treatment was 
not effective, and these rationales 
point to areas of future direction for 
this research initiative. First, regarding 
the delivery of the treatment, a pre- 
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recorded telephone call may no longer 
be the best mode for reaching these 
students. Although telephone contact 
with 449 of the 500 students in the 
treatment group resulted either in a 
live answer or a redirect to an 
answering machine, I do not know who 
(e.g., parent, student, sibling, or other 
relation) received the message and if 
that message was properly conveyed 
to the student. Other modes of contact 
(e.g., email) will be part of future 
studies to determine if modifying the 
mode of delivery improves the test day 
attendance rate of these students. 

Second, regarding the timing of the 
treatment, lower-income students (who 
are the target population for this study) 
are more likely than their peers to face 
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